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ABSTRACT 

This paper examines the evaluation of instructional 
materials from three viewpoints. First, an analytic system is 
described vhich has been found useful by practitioners and 
researchers for examining the instructional design of materials. This 
system can contribute to selection among competing materials and to 
improved use of materials in the teaching ^learning process. It is 
suggested that instructional materials be examined for the quality 
and completeness of their instructional design and their 
compatability (fit) vith the curriculum design of the school. Second, 
a broader syst^z is described which holds promise of analyzing, 
developing, and maintaining a reference file on the thousands of 
instructional product 3 now on the market. Schools would be able to 
retrieve from the reference file information on the particular 
instructional design constructs of materials. Third, the significance 
of materials evaluation in meeting the goals of two different 
curriculum designs is discussed. A selected analysis of the 
literature and the major emphases in materials evaluation is 
presented . (A uthor ) 
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ABSTRACT 

This paper examines the evaluation of instructional materials from 
three viewpoints. First, an analytic system is describee} which has 
been found useful by practitioners and researchers for examining the 
instructional design of materials. This system can contribute to 
selection among competing materials and to improved use of materials 
in the teaching-'learning process. It is suggested that 
instructional materials be examined for the quality and completeness 
of their instructional design and their compa tab il i t y (fit) with the 
curriculum design of the school. Second, a broader system is 
described which holds promise of analyzing, developing, and 
maintaining a reference file on the thousands of instructional 
products now on the market. Schools would be able to retrieve from 
the reference file information on the particular instructional lliljw 
design constructs of materials. Third, the significance of x SoSl'lj; 

materials evaluation in meeting the goals of two different <So 2|igd|| 

curriculum designs is discussed. A selected analysis of the ^dls^aJ^jJ^K 
literature and the major emphases in materials evaluation is 5l;5;5<4n<>uoz 
presented • - 

***** 




Instructional materials are essential ingredients in educational 3 ^S^oj.^ 

programs. Since perhaps 70 percent of a student s classroom work fofSftSS 
involves these materials, their influence outweighs the minor cost 
they impose on the school budget. Despite their significance, 
however, the systematic attention given them by researchers has been 
very limited compared with other classroom phenomena such as 
teaching strategics, student behavior, and environments. 

The major study of instructional materials was completed 20 
years ago and reflects the heavy emphasis on the textbook as the 
principal instructional material (3). Since that time, two lines of 
work have given attention to the selection of content and the 
instructional design of a variety of educational materials. 

Analyzing Content 



^ In the analysis of content, typically, the researcher compares the 
aX^ content against outside judgments of what is appropriate. For 
example, in a typical study. Brown (1) examined the scientific 
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principles presented in selected junior high school science 
textbooks and made judgments, on the basis of research, on which 
principles should be taught in a three-year Junior high school. 
Other examples of studies that have assessed subject content for 
inclusion have been done in conservation education (9), on Latin 
America (8), and on earth science (10) • Over the past decade, it 
has become common to examine instructional materials for their 
portrayal of certain selected groups, minorities, and women. One 
variant of this analysis has been the examination of instructional 
materials for realistic depiction of urban life (2) and for use with 
inner city students ( 1 1 ) . 

The worth of content, its inclusion or exclusion, is a 
long-standing problem that has divided communities, politicized 
school boards, and brought great pressure to bear on superintendents 
and faculty. For an example of responsible criticism of content 
selec tion • see the Wall Street Journal , editorial July 21, 1975 on 
•*MACOS and Moral Values." Irresponsible and ax-grinding criticism 
is easy to come by. (For numerous examples, see The Censors and the 
School s by Jack Nelson and Gene Roberts Jr. Boston: Little, Brown & 
Co., 1963.) 

Arguing the merits of content primarily selected on judgmental 
criteria set by professional opinion or an interpretation of 
community norms has not been a productive enterprise for 
instructional-materials selec tion committees . 

Professional societies have given much attention to the problem 
of content selection and inclusion in numerous statements of 
recommendations. These recommendations have generally been used as 
guidelines by producers and selection committees. As a rule, the 
content recommended and included by producers does not run the 
danger of offending a core of widely accepted values and public 
moral standards. 

Studying the Instructional Design of Materials 

The second basic line of work has focused on the the instructional 
design characteristics of the materials (4, 13). A variant of the 
system developed by Eash and now used by the Educational Products 
Information Exchange in its nationally oriented evaluation system is 
described on page 3. Numerous field tests with practitioners (5) 
demonstrate the usefulness of the system for examining the 
instructional design of the materials for construct weaknesses, 
contradictions, and input required to make it an effective learning 
package and to teach users the intricacy of the instructional design 
within the materials, its purpose and learning thrust. 

Our experience indicates that many learning packages are 
Incomplete, inconsistent in their instructional design, and leave 
much to the ingenuity of the teacher. Producers who have expended 
considerable sums on the development of comprehensive instructional 
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design frequently find it not living up to their original promise 
due to the lack of mastery of the design by the user (6> • For the 
proper use of materials, our experience at the Office of Evaluation 
Research at the Universiy of Illinois at Chicago Circle strongly 
recommends that school systems hold materials evaluation workshops, 
on new materials to preclude misuse of instructional materials and 
to instruct teachers in the accompanying instructional design. 
Moreover, in these workshops teachers become aware of the need for 
supplementing the instructional design if it is inadequately 
provisioned • 

Four Constructs of Curriculum Design 

The system for evaluating instructional materials is based upon four 
constructs of curriculum design: objectives, organization of 
materials (scope and sequence), methods of instruction, and 
evaluation^ A number of components are examined under each of these 
constructs. The questions under each construct shown below are 
suggested for use as guides by practitioners in assessing 
instructional materials. 

I. OBJECTIVES 

A. What is the nature of the general goals of the material 
stated? 

B. Are specific objectives stated for teacher use? 

C. If neither of the above are stated, list what you believe are 
the intended objectives of the material. 

D. What are the main emphases in the objectives? 

E. On the scale below, rate the objectives of the materials. 
Please place an "X." on an exact po,int. 

ObjzctLvQA not ode- ObjtoZivu qI\)<l qJLzoji dJouLo,- 

iuJi to a taackoJi. 1 2 3 4 5 6 ^n^tAucZion and 

M^iul ioK a taachoA. 



II. ORGANIZATION OF MATERIALS (SCOPE AND SEQUENCE) 

A. What is the scope of content covered in the materials? 

B. How is the scope of the materials organized? 

C. Is there a specified sequence in the material? 

D. What is the basis for the suggested sequence? 

E. On the scale below, rate the scope and sequence of the 
material. Please place an "X" on an exact point. 



Scope Inadnqaata, Scopa ade^quate. ioK QHjOida ok 

OK Incomplatz. ^ ^ 4 :> 5 j^^jj^y lntQMAQZatQ.d and planned. 
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III. METHODS OF INSTRUCTION 



A. What method or methods of instruction are suggested? 

B. What role is emphasized In the method: teacher, pupil, or 
both? 

C. What are the specific features of the method or methods 
recommended? 

D. Is the suggested method one that requires the teacher to do 
extensive prior preparation or participate in specific 
training? 

£• On the scale below, rate the methods of instruction. Please 
place an "X" on an exact point. 



Wo moJthod^ 6aggz6te.d ok VoJiu ccULtiulty davalopad 

imptizd that oAa hdZpial 1 2 3 4 5 6 both 
to a taachoJi. tzaciiVi and pupil. 



IV. EVALUATION 

A. What test materials are included for the student's and 
teacher's use? 

B. Are the test items adequate for informing a teacher of 
students' progress toward the instructional objectives set 
for the materials? 

C. What do the tests measure? 

D. Is there information on the tests' reliability and validity? 

E. Is there any information from the producer on how the 
materials were tested with students when they were being 
developed? 

F. On the scale below, rate the evaluation components of these 
materials and the evaluation of the materials by the producer 
as they were developed. Please place an *'X" on an exact 

po int . 

t4o tdt^t mateJUaU ok 6ug^ A totde Kongo, oi tut matdAA^aU 

gzhtad uizckt on ^tudunt and ^vaZuation 6ugga^tlon6. 

IzoAnlng incZudtd. He data ^ ^ ^ ^ . . EvaZaatlon data on ^oJU tut 
on the. e.valuat.ion o^ mot- - j ^ d o ^^^^^^^^ matVLiaJU 

zKLoU by tha pKoduaoA. Indbxdad. 



V. TOTAL RATING OF THE MATERIAL 



A. Draw up a brief statement on how these materials compare with 
those currently being used in your curriculum. 

B. On the scale below, rate overall potential effectiveness of 
these materials. Please pi ace an ''X" on an exact point. 



^a;t2M.aJU contain many 
maknu^u tn tn^t/iuct- 
tonat dutgn. Vtif^catt 
to (ue., expe>t6^ve, tn- 
\2AL0K ioK IzoKning. 



VoMf 6tKong In aJUi axaoj^ oi 
dutgn. StKong potential to 
develop a ijuldo. va/UoXy o<( 
laaAnlng^. Oi high intojiut to 
to^achoM and pupils. VoAif 
co^t e^iie^atLve.* 
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VI. CONSTRUCT RATING PROFILE 



A. After completing the instrument, fill out the following 
profile for the evaluated material: 



RATING 
6 
5 
4 
3 
2 
1 

CONSTRUCT 



II 



III 



IV 



RATINGS 
6 
5 
4 
3 
2 
1 

CONSTRUCTS 



TABLE 1 

A Profile of Two Mathema-.ics Series 



X* 

X- 



II 



III 



IV 



, = Series I 
= Series II 
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Table 1 on page 5 is a profile of two mathematics series. As 
It.-shows, both series have their strengths and weaknesses, and If 
.either should be adopted In use, the teacher wouH have to provision 
some of the constructs to make the series more adequate teaching 
tools, A school system may want to balance the strengths against 
these extra costs or It may decide the extra costs are worth 
ab sorblng to gain the strengths contained In the series. In any 
case, a far more rational basis exists for the selection of 
Instructional materials, which Is an advantage to bo*"^i student .and 
teacher as well as the producer. 

Dissemination Network of Materials-Design Analysis 

With well over 300,000 different Instructional materials now on the 
market, each of which Is competing for the carefully husbanded 
school funds available for acqulstion of new materials, which are 
estimated at ,97 of one percent of the total school budget ( American 
School Board Journal , 1975), there must be a systematic way to 
analyze the design of Instructional materials. No one school system 
or state department of education can tackle the task. Despite this, 
several states with textbook adoption laws now require some form of 
evaluation of materials that have been submitted for possible state 
adoption. Most school districts devise their own evaluation forms; 
on the school building level, the forms undergo further changes. 
Hence, Information about Instructional materials Is collected In 
bits and pieces, filed away, and never exchanged with other schools 
whose personnel could be saved the time-consuming task of evaluating 
the same materials. Even If the Information were exchanged, there 
Is a language barrier, for there are no agreed-upon categories for 
selecting and coding the information for storing and sharing. 

The instructional materials-design analysis instrument cited 
previously might serve a storage and sharing purpose. Underlying 
the instrument is a conceptual schema of curriculum design. 
Instructional materials selected for use in the educational program 
become the media through which the curriculum is translated Into 
Instruction. Therefore, understanding the design of the materials 
helps the decision maker select the materials that are appropriate 
for a given curriculum. If greater numbers of educators learn to 
use the materials-design analysis Instrument, the selection process 
could avoid the usual difficulty of political pressure, publishers' 
flashy presentations, peer arm-twisting, and subjective decision 
making. 

This section reports on a national effort to establish a common 
language of materials-design analysis and to share the data on a 
wide scale. The Educational Products Information Exchange Institute 
(EPIE), a nonprofit organization, was established in 196 7 under a 
charter (7) by the Regents of the University of the State of New 
York: 



The purpose of EPIE Institute is to provide its members with 



information and counsel based on impartial, independent studies 
of availability, use and effectiveness of educational materials, 
equipment, and systems. All EPIE's services are designed to 
facilitate the making of informed rational evaluations of 
specific products by the educational consumer. 

EPIE uses an adapted form of the Eash instrument to analyze 
materials and presents the results in the EPIE publication EPIE 
Reports . At first, only a handful of evaluators knew how to apply 
the constructs to design analysis. Under EPIE's direction and 
through the sponsorship of state departments of education and local 
regional educational offices, personnel in key ma t erial s-' sel ec tion 
positions participated in two-^and-^one-^half-'d ay training sessions on 
materials-'design analysis and a day-long follow-'up session one month 
later. This was the first step beyond publication of the EPIE 
Reports to introduce educators to a common analysis schema, and the 
first effort to make the instructional mater ials-^design analysis 
instrument a feasible format for school personnel to use in their 
own materials evaluation tasks. To date, educators in states across 
the country and in Canada have completed the training and are 
qualified as analyzers. The analyzers send EPIE a copy of their 
analyses. These are synthesized and become part of the common data 
bank for disseminating information about instructional materials 
through the EPIE Reports . In the course of the training sessions, 
the initial instrument has undergone .everal changes to make it 
responsive to. the kinds of data most meaningful to the users. 
Training sessions have been conducted in the states of Pennsylvania, 
Connecticut, New Jersey, Florida, California, New Mexico, New York, 
Texas, and in Canada. 

As a next step in establishing a dissemination network, EPIE 
organized a task force. Through the employment of the task force 
concept, EPIE obtained systematic data in selected content areas to 
share with a wide audience. The latest task force devoted to 
analyzing current elementary mathematics series assembled for five 
days. The team consisted of EPIE-' trained analyzers from state 
departments of education, school systems, a special group of 
mathematics educators selected by the National Council of 
Mathematics Teachers (who underwent EPIE materials-'design analysis 
training), and EPIE staff and materials-devaluation consultants. 
Current mathematics series from the major publishers were analyzed. 
The series were analyzed independently by an educational generalist 
and a mathematics educator using the latest version of the EPIE form 
for analyzing instructional materials. These independent analyses 
were synthesized by an EPIE staff member or the evaluation 
c onsul t an t . 

The information is now being disseminated in an EPIE Report . 
School personnel across the country have available analyses of 
nearly two dozen major mathematics series, described in a common 
language, that can be compared with each other. Instructional 
materials-^selection decisions can now be predicated on the "best 
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fit," that is, sell '^♦'ing the mathematics series whose design best 
fits the school sysi 's mathematics curriculum or its philosophy of 
mathematical educatii • Task forces have also been organized for 
analyzing extant reed ng and social studies texts that have been 
reported in past issue of the EPIE Report . 

The use of a common language to describe the design features of 
instructional materials is used as a form of communicating complex 
concepts in operational terms understood by an ever-'inc r easing group 
of educational decision makers. Where EPIE Repo r t s describing a 
textbook series formerly covered eight to a dozen pages, analyses 
are now compressed into two EPIE Report pages. Expository 
statements are reduced to operational descriptors. 

Plans in the immediate future include coding the descriptors 
for computer storage. As greater numbers of analysts throughout the 
country send their reports to EPIE, the analyses can be synthesized 
and the descriptors coded and stored. Upon request, educators will 
be able to obtain from EPIE computer printouts of instructional 
materials desi^ i analyses. In the future, school personnel can ask 
for information about specific materials, materials in a given 
subject area, or materials having certain design features. 

Analysis of the design of instructional materials does not 
remove from school personnel the final responsibility of evaluating 
and selecting materials. Evaluation becomes a "best fit" task. 
Educators must understand their own curriculum and fit the design of 
the materials to their curriculum designs. 

Materials Evaluation in Individually Guided Education and Open 
Education 

Materials evaluation is particularly important in the very recent 
curriculum designs called Individually Guided Education (IGE) and 
Open Education, which represent, respectively, a behaviorally 
oriented system and a humanistically oriented plan (6 and 14). 

Individually Guided Education, developed by the University of 
Wisconsin Research and Development Center for Cognitive Learning, is 
a system of organizing the school program to make it more responsive 
to the needs of individual children. Based on a model of 
instructional programming for the individual child, criterion-* 
referenced tests, and observation schedules, IGE calls for the 
assessment of each child's entry skills, use of a management plan 
for grouping and instructing children, and a plan for monitoring 
each child's progress. However appealing though the model may be, 
the IGE materials for student^ are not at all extensive, and many 
are still being field tested. Consequently, the school staff must 
select and use commercially r.-^"ilable materials that are judged 
congruent with, or adaptable to, IGE goals and procedures. Thus, 
IGE teachers especially those lust starting the program (as many 
are now doing around the country) must hv materials ^valuators and 
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highly aware of such design constructs as scope and sequence that 
determine the compatablll ty of sets of materials with one another 
and with the IGE system* 

Open Education, especially In Its more authentic, original 
form, puts an even greater burden on the materials-devaluation skills 
of the teacher who, with her students, jointly selects or fashions 
the materials of learning* Class sets of texts are avoided, and the 
book most suitable for a particular child at a particular time is 
selected* In many cases, a book written by one child is studied by 
another* In other instances, ^ child may read and discuss his or 
her writing with other interested children* Similarly, the teacher 
and children bring in natural materials for scientific study or 
craft materials that serve as the basis of lessons* 

In traditional education, adherance to a reading series and 
comparable media would insure appropriate coverage, sequence, 
difficulty, and other design constructs* In Open Education, these 
responsibilities fall on the teacher and child; obviously, awareness 
of the design issues mentioned in the previous sections are critical 
not only for classroom success but also for explaining purposes and 
procedures to parents* 

Conclusions 

Changing curriculum designs put ever greater pressure on staff and 
teachers to be knowledgeable about the quality and potential of the 
design of instructional materials* Increasingly, the teacher 
becomes a key figure in an instructional management system in which 
materials have a specific, as opposed to a general, function* To 
bring to fruition the promising trends in curriculum design 
described in this paper will require an in-ddepth knowledge of 
materials* Some limited assistance is becoming available 
nationally, but the primary responsibility rests with the faculty in 
the local school unit* The Instrument presented in this paper is 
one approach to meeting the need for assessment of the instructional 
design of materials and the promotion of in-'Service education for 
improved use of materials* 
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